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Abstract 

Much of the recent research on homework focuses on using online, web-based, or com¬ 
puterized homework systems. These systems have many reported capabilities and bene¬ 
fits, including the ability to randomize values, which enables multiple attempts by a stu¬ 
dent or to reduce academic dishonesty. This study reports on the impact of using random¬ 
ized versus static homework values on student learning as measured by student perform¬ 
ance on similar exam questions. Results indicate randomization can have a significant 
practical and statistical impact on improving exam scores, can reduce the rate of cheating, 
and can make homework a more accurate measure of student ability as the difference be¬ 
tween homework scores and exam scores is reduced. 

Keywords: Randomized homework, academic dishonesty, improved learning. 


When looking at the history of research into the efficacy of homework for learning, the 
only thing clear is how muddy the results are. The concern with whether homework is 
helpful or not to student learning has been researched for decades without resulting in an 
agreement on its pedagogical value. Cooper (1989) is an important early review paper on 
the efficacy of homework for improving learning. In well-designed studies that controlled 
for homework versus no homework, though, only 70 percent showed a positive benefit of 
homework with high school students receiving more benefit than elementary students. 
Follow-up studies (Cooper & Valentine, 2001; Cooper, Robinson, & Patall, 2006), 
meanwhile, have found the grade level effect was still pronounced, that homework gener¬ 
ally has a positive or at least nonnegative impact, but that there is still “no strong evi¬ 
dence” (Cooper et ah, 2006) between the homework and achievement link. The authors 
suggest, after all these years, future research is still needed as most research into home¬ 
work has used overly complex models that cause potential confounding of results 
(Trautwein & Roller, 2003; Cooper et ah, 2006). 

While the majority of research into homework focuses on K - 12 students, results at the 
college level are similarly conflicting. In my area of business education, a frequently 
cited study by Rayburn and Rayburn (1999) in a managerial accounting course showed 
consistent improvement on exam performance for those completing quantitative home¬ 
work compared to those who did not. Peters, Kethley and Bullington (2002), meanwhile, 
reports that requiring homework in an operations management (OM) course did not im- 
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prove results on quantitative exam problems and may have had a slight negative effect on 
overall perfonnance. Peters et al. (2002) surmise that in their study students in the 
graded homework group adopted a “zero-sum game” mentality at the end of the semester, 
figuring they had already put in enough of their effort into the course. Furthermore, they 
note that the exam problems were multiple-choice and may have been too different from 
the quantitative, problem-solving homework structure to detect performance differences. 
I found the Peters et al. (2002) results intriguing because I teach OM at a regional campus 
in the mid-west where in our program OM is an upper-division, core business course tak¬ 
en by all business majors. I had been assigning homework problems in the course, yet too 
often, performance on exams in my classes would fail to impress. 

Interestingly enough, while the case for homework at all levels is still far from clear, 
much of the homework research since Peters et al. (2002) has introduced yet another fac¬ 
tor into the equation: technology. Using or comparing online, web-based and computer¬ 
ized homework systems for quantitative problems to traditional approaches has gained 
favor. Computer-based systems can be useful for instructors (Heizer, Render, & Watson, 
2009), can positively impact student perceptions and efforts in a course (Smolira, 2008), 
can be particularly impactful for low-motivated students (Peng, 2009), and can help stu¬ 
dents self-regulate themselves for learning (Hauk & Segalla, 2005). For example, ran¬ 
domized homework values can make copying answers more difficult or allow multiple 
attempts, where each attempt is different so memorization does not occur. Then, immedi¬ 
ate feedback can be given through automatic grading. With capabilities such as these, 
positive impacts on learning should follow, right? 

Unfortunately, conflicting results still appear to be the nonn. In math and science, for ex¬ 
ample, several studies (e.g., Affouf & Walsh, 2007; Zerr , 2007; Burch & Kuo, 2010) 
find positive impacts of utilizing this homework technology while others (e.g., Bonham, 
Deardorff, & Beichner, 2003; Allain & Williams, 2006; Demirici, 2007; Kodippili & 
Senaratne, 2008) do not. 

In my area of business education, Palocsay and Stevens (2008) investigates the effective¬ 
ness of web-based homework in teaching undergraduate business statistics. While some 
performance improvements were seen, the variability in instructors and student ability led 
to the conclusion that homework technique, including data randomization, had little im¬ 
pact. 

Heizer, Render, and Watson (2009), on the other hand, report positive impact in using 
web-based systems in their graduate and undergraduate operations management classes. 
On homework assignments, scores improved by 17 to 35 percent through multiple at¬ 
tempts using the web-based Prentice-Hall Grade Assist (PHGA) program. In an MBA 
program, improvement for the group using the PHGA system over the control group as 
measured by final grade average was nearly 10 points. In addition to improvements in 
student learning, the authors note gains in efficiency and integrity in grading along with 
the possibility to lessen academic dishonesty due to randomization of questions and data 
values. 
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Rapid development of commercially available homework systems is welcomed even if 
the jury is still out on how to best use them. For example, like Peters et al. (2002), I was 
not seeing benefits of using homework on student performance. Unlike Peters et al. 
(2002), though, my homework problems were very similar to the exams and yet too often 
students who earned a high score on a homework assignment would suffer a large per¬ 
formance drop, or gap, on similar exam problems. For example, students averaged nearly 
a 20 point drop on the exam for the same problem type that they had seen on the home¬ 
work assignment. This large drop occurred even though many students had earned a 100 
on the homework. Not only did these results bring into question the value of requiring 
graded homework, but it pointed to the likelihood that some students were simply copy¬ 
ing homework answers. 

When I surveyed students about academic dishonesty on assignments as part of the ano¬ 
nymous end-of-semester student evaluation process, though, only a couple students ad¬ 
mitted seeing others copy work and none admitted it for themselves. Students may not 
have been comfortable acknowledging the existence of copying answers but the perform¬ 
ance data indicated otherwise. 

Indeed, research into academic dishonesty has shown more than 40 percent of students at 
all institutions acknowledged copying homework assignments and nearly 80 percent ac¬ 
knowledge seeing or participating in some kind of cheating (McCabe & Trevino, 1993). 
How prevalent academic dishonesty is for business students compared to other college 
students is a matter of contention, though. A study by Iyer and Eastman (2006) indicates 
it is lower while McCabe, Butterfield, and Trevino (2006) show a higher rate. Overall, 
contextual influences on academic dishonesty find factors such as membership in frater¬ 
nities and sororities, peer pressure, gender, year in school, pressure to achieve, individu¬ 
alization, and perceptions among students about the acceptance and rate of cheating are 
seen to be important (Davis, Grover, Becker, & McGregor 1992; McCabe & Trevino, 
1997; Pulvers & Diekhoff, 1999; Whitley & Keith-Spiegel, 2002). 

While understanding these influences is instructive, for the individual teacher, how to ef¬ 
fectively minimize dishonesty so true learning can be achieved is arguably most impor¬ 
tant. In my case, moral appeals and denoting the extent to which students were allowed to 
discuss the assignments (Whitley & Keith-Spiegel, 2002) were already being made. Fur¬ 
thermore, the assignment values were being changed for each class (Whitley & Keith- 
Spiegel, 2002) and yet these efforts were being met with limited success. Hence, I was 
motivated to develop and implement a randomized data approach with the hope it would 
help. With no commercial systems meeting my exact needs, I developed my own system 
using capabilities found in Microsoft Office. 

In summary, the primary motivation behind the randomized homework approach is to 
improve learning as measured by student ability to solve similar problems under con¬ 
trolled exam conditions. A major concern was that a large performance gap existed for 
some students in the course where exam scores on similar problems were much lower 
than on their homework. This implied that whatever efforts students made or interactions 
they had with static homework values; it was not being translated into individual per- 
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formance on exams. As a result, the potential benefit homework could have on learning 
was not being fully realized. A more targeted design than employed by others (e.g., Pe¬ 
ters et ah, 2002; Palocsay & Stevens, 2008; Heizer et ah, 2009) was developed to avoid 
confounding results (Cooper et ah, 2006). In particular, the design required not only look¬ 
ing at changes in the overall performance on exam problems and homework scores but in 
matching individual student homework scores directly to their related exam problems. 
While the study reported in this manuscript has a design that extends the important foun¬ 
dation work of Peters et al. (2002) and Heizer et al. (2009) in the operations management 
area, it should have applicability to other quantitative, problem-solving courses and be 
independent of implementation platfonn. 

The remainder of the paper details my efforts to implement and study randomized home¬ 
work. First, the hypotheses developed for the investigation are discussed, which is fol¬ 
lowed by experimental design considerations. Then, details on the homework assign¬ 
ments, including generation and grading considerations are covered. Finally, the results 
are presented and discussed, including limitations of the study and future considerations. 

Hypotheses 


HI: Student Exam Performance 

The first objective is to improve student performance on exam questions of the same 
problem types as found on the homework assignments. This objective is measured via 
hypothesis HI, which states that student perfonnance on exam problems will improve 
after implementing randomized homework assignments. 

Exams in the course included multiple choice questions, calculation problems with re¬ 
lated analysis questions, and essays. The multiple choice questions are almost exclusively 
qualitative in nature and serve to monitor general concept understanding and to check 
completion of reading assignments. The majority of point values on exams, however, is 
associated with calculation problems and related analysis questions. 

To measure HI, only the exam calculation problems and analysis questions related to a 
homework assignment are evaluated. This avoids confounding performance results with 
concepts and question structures unrelated to homework that potentially impacted results 
of others (e.g., Peters, et al., 2002). 

For example, the homework assignment for statistical process control (SPC) included an 
x-bar and r -chart application that had to be identified and calculated by the student from 
the descriptive infonnation on the assignment. On exam two, a similar set of SPC prob¬ 
lems were posed. If these problems represented 60 points on an exam, and a student 
earned 54 points, an exam performance of 90 percent was recorded for this study. Details 
on the structure of exam performance determination, especially related to efforts for 
maintaining consistency of measures between the control group and post-implementation 
students, are found in the section on experimental design. 
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It should be noted that the homework, whether randomized or not, always precedes the 
corresponding exam by at least two class periods. This is intentional so students can re¬ 
view their performance and seek help in advance of the exam. Furthermore, students are 
encouraged to bring two copies of their submission to class when the homework is due 
because after collecting their submission I solved it (or a randomized case of it) in class 
in order to provide some immediate feedback, which has been shown to improve student 
performance (Yourstone, Kraye, & Albaum, 2008; Smolira, 2008). Graded assignments 
were returned to students at the next class meeting. 

H2: Student Homework Performance 

While increasing exam perfonnance is the desired effect, the overall performance on the 
homework assignments is expected to decrease. This represents the second hypothesis, 
H2. With randomized homework assignments, the process for students to check their 
work with others in ad hoc fashion is more complex and requires significant understand¬ 
ing and effort. Students who forget or procrastinate do not have an easy way to simply 
acquire the correct answers, which should lead to lower homework scores as postulated in 
H2. Related to this, it was believed that much of the decrease in homework scores would 
be from students who achieved a perfect homework score before randomization but not 
because of their own understanding. Therefore, the percentage of students earning a per¬ 
fect score on homework assignments before and after randomization is monitored and 
reported. 

H3: Reducing homework versus exam performance differences 

Perhaps the greatest motivation for developing randomized homework assignment values 
resulted from the observation that there were too many students getting high scores on 
homework assignments that subsequently performed poorly on the same material on ex¬ 
ams. A large gap between a student’s homework and exam performance certainly brings 
into question the value a homework assignment had for learning. Therefore, in addition to 
improving the overall exam scores from HI, an important objective is to reduce the indi¬ 
vidual differences between student exam and homework performance. 

The third hypothesis, H3, states that the gap between homework and exam performance 
on a student-level basis will decrease after implementing randomized homework assign¬ 
ments. In effect, there will be a significant reduction in students scoring very well on 
homework assignments that are not being able to perform well on similar problems dur¬ 
ing the exam. 

H3 is measured by looking at the difference between each student’s homework score and 
their exam score on the same problem type. The homework score for each student was 
readily available as it was recorded directly in the grade book while the student’s exam 
score was calculated as in HI. The gap between the homework score and the relevant ex¬ 
am performance was then calculated. For example, a student who earned a 100 on home¬ 
work but only scored 90 percent on the exam material had a difference score of negative 
10. H3 differs from the first two hypotheses in that HI and H2 are measured from an 
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overall group performance level instead of at the student level and it is expected to have 
strong identification power. 

Experimental Design Considerations 

Several important issues were considered in the design of this evaluation. These included 
adopting an appropriate experimental design structure, maintaining assignment integrity, 
and minimizing grading inconsistencies that could bias results. The efforts addressing 
these issues are discussed next. 

Experimental Design 

To evaluate the hypotheses, a pre-test/posttest experimental design is employed in this 
study. The pre-test period represents a control group of 63 students with static homework 
data while there are 94 students who received randomized values in the implementation 
phase. In all statistical tests, a conservative assumption of unequal variances was used 
even when an equal variances assumption could be justified. 

There are three homework assignments: forecasting, SPC, and inventory; each corre¬ 
sponding to a different exam through the semester. Forecasting was on exam one, SPC on 
exam two, and inventory management on exam three (the non-comprehensive final ex¬ 
am). Similar homework assignments and exam questions were used in each phase and 
special precautions were used to ensure consistency and to maintain assignment integrity 
as will be discussed below. 

Maintaining Assignment Integrity 

Maintaining assignment integrity is important in any teaching situation but is critical in 
an experimental study like this. The actual homework assignments were modified each 
semester regardless of whether randomization occurred or not. The problem types are the 
same but the actual problems differ and strict control of the questions themselves is not 
possible, nor attempted, given that students complete homework outside of the classroom 
and can easily pass copies along. Even when static values were used in the homework, 
the actual values were changed semester to semester. Exams, though, were controlled 
carefully to ensure no copies were stolen. Integrity was accomplished by not only count¬ 
ing the copies but in not recording student grades until after the exam was reviewed in 
class and all exams were verified as returned. 

Minimizing Inconsistencies in Grading 

Another important concern was controlling the consistency of exam and homework grad¬ 
ing. Rubrics played an important role here. Effort was made before the control phase be¬ 
gan to standardize grading where common mistake areas were identified and point deduc¬ 
tions were determined. These rubrics were used for both homework and exam grading to 
minimize inconsistencies from student to student and over time. 
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Homework Assignments 

This study incorporates three different homework assignments in the randomization ex¬ 
periment. Each assignment, which corresponds to a different exam during the semester, is 
detailed next. Then, information on assignment generation and the mechanics of grading 
with randomized data is discussed. 

Forecasting 

The first homework assignment of the semester relates to forecasting. Students are re¬ 
quired to develop forecasts where each student has different data series values and vari¬ 
ability. Three forecasting methods are required including moving averages, weighted 
moving averages, and exponential smoothing. Forecast evaluations are made and error 
measures are calculated, which include bias, mean absolute deviation (MAD), and mean 
square error (MSE). Students must ultimately choose the best overall forecasting method, 
provide an explanation of why it was chosen, and answer additional questions related to 
the forecasting process they just completed. This is a significant effort for students, espe¬ 
cially early in the semester. The concepts themselves are not particularly difficult but the 
sheer number of new issues is intimidating to students. 

Statistical Process Control 

The second homework assignment relates to statistical process control (SPC). In this as¬ 
signment, an organizational application is described and data given. Students must iden¬ 
tify the correct chart(s), calculate the control limits, and develop the graphs. They then 
must answer related questions, such as to identify if the process is in-control or not. In its 
current fonn, the application is the same for all students, a variable data situation necessi¬ 
tating x-bar and r-charts, but the values and decisions vary in the randomization phase. 
For example, one student may have a process that is in-control while others have out-of¬ 
control process data. 

SPC is typically the most conceptually challenging section of the course for students and 
considerable time is spent reviewing the probability and statistics theories that underlie 
the charts. While the textbook is used for theory and understanding, students use The 
Memory Jogger II (Brassard & Ritter, 1994) for implementation. This pocket reference 
has a decision tree students use to select the correct chart, a step-by-step guide for control 
chart implementation, formulas, constants, and information on how to interpret the charts. 
The Memory Jogger II is a handy reference book that students are allowed to use on ex¬ 
ams too. SPC related questions are found on the second exam of the semester. 

Inventory Management 

The third homework assignment relates to inventory management. As with SPC, a de¬ 
scription of an organizational setting is provided to students along with randomized data 
for costs and usages. Students must identify the correct inventory model, calculate items 
of interest such as order quantity, reorder point, and total cost, and then answer related 
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questions. Inventory management is one of the last major topics of the semester and it 
corresponds to questions on the final exam. 

Homework Generation 

The randomized assignments in this study are implemented using Microsoft Office and 
were developed by the author. It should be noted that textbook publishers have tools that 
can accomplish randomization that might suffice for other instructors. 

In this implementation, students are sent an Excel file as an attachment to an email or 
given a file link to download. When the file is opened, only one spreadsheet is visible in 
the workbook. All other sheets and programming are hidden and password protected. A 
Visual Basic for Applications (VBA) macro automatically begins, which opens a dialog 
box prompting the student to enter his/her name and the last four digits of his/her student 
identification (ID) number. Error checking routines are employed to ensure all four val¬ 
ues are entered and that they are indeed numbers and not letters or other characters. 
Those interested learning more about coding in VBA should find Walkenbach (2007) is 
an accessible, yet comprehensive, resource. 

The ID is used to generate the randomized values for each student where a minimum of 
1000 cases are created for each. The student ID is then scrambled before being used to 
generate data, where a different ID pattern is used for each assignment to make it im¬ 
probable that two students could have the same or similar values for multiple assign¬ 
ments. 

Each case for an assignment has both a systematic adjustment and a random error com¬ 
ponent. For example, in a forecasting assignment, an initial pattern was developed and a 
systematic adjustment was used to shift the cases in scale, going from several hundred 
units in some cases to several thousand units in others. Then, a random adjustment was 
added to the individual values so the pattern would show variability and not simply be a 
shift that could be readily identified through graphing. 

This approach facilitates the ability to have differing decisions from student to student 
based upon the specific case data. As a result, not only do the values differ but the analy¬ 
sis and conclusions can vary too. This was explained to students so they would under¬ 
stand it, get over it, and focus on making decisions consistent with what the data indi¬ 
cated was correct for their case without regard, or fear, for what others may have been 
seeing. 

Grading of Student Assignments 

One of the significant concerns with using randomized assignments for student home¬ 
work is how to grade efficiently when the values are different for each case. Indeed, at 
the 2009 Decision Sciences Meeting in New Orleans, several of the publisher offerings 
for randomized or internet-based homework systems presented in break-out sessions em¬ 
phasized instructor efficiency given automatic grading capabilities. While the prime mo¬ 
tivation in this study is to detennine if randomized assignment values do indeed improve 
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student learning without regard to implementation platform, automatic grading via pub¬ 
lisher offerings is an added bonus for instructors with large section sizes. 

On my campus, section sizes are kept below 35 students, so maintaining the connection 
and feedback obtained from personally grading student assignments as part of teaching 
reflection is still possible. To facilitate the individualized grading key, a solution work¬ 
sheet was created in the Excel workbook for each assignment. This sheet initially was 
used for error checking the systems and for reviewing the development of the student 
cases in each assignment. This worksheet was deleted from the file given to students but 
it was used in conjunction with the roster and simple looping logic to develop customized 
solution sheets for each student. These were aggregated into a class key. 

The main grading efficiency loss came from the fact that memorizing the correct answers 
was not possible. However, with good grading key structure, just a little extra time, per¬ 
haps 20 to 25 percent, was needed. Of course, online and publisher-based grading tools 
will further reduce or eliminate efficiency concerns for those interested in using random¬ 
ized assignment values in their courses. 

Results and Discussion 

The first hypothesis, HI, tests if student performance on exams improved during the ran¬ 
domized homework phase of the study. In figure 1, it is seen that exam performance in¬ 
creased by 3.83 (p-value = .083), 3.63 (p-value =.038), and 2.65 points (p-value =.188) 
respectively on each of the three exams. Recall that the first exam covered forecasting, 
the second included SPC, and the third covered inventory management. Therefore, ran¬ 
domized homework has a statistically significant impact on the first two exams but less of 
a statistical effect on the third. 

From a practical standpoint, it is encouraging to see improvement range from one-quarter 
to nearly four-tenths of a letter grade for students in the randomized homework group rel¬ 
ative to the static homework students. Even from a theoretical standpoint, the gain 
achieved, as measured by Hake’s gain statistic (Hake, 1998), g, shows that between 11 
percent and 17 percent of the total gain possible is achieved simply by implementing ran¬ 
domized homework values. As an instructor, I was pleased with these results, especially 
considering that during the static data phase, the range of exam averages across the three 
exams was only 1.37 points, roughly one-half the impact seen from randomization. In 
other words, randomizing homework values led to a larger impact in student performance 
than a change in topics—from forecasting to SPC to inventory—had during the static da¬ 
ta phase. 

It was not initially clear why the forecasting and SPC topics had a greater improvement 
than the inventory material; however, viewing the results in terms of the end-of-semester 
gaming phenomenon, where students seek to maximize their grades across courses as po¬ 
sited by Peters et al. (2002), sheds light on the issue. For example, students who have 
done well in a course and built up a cushion going into the final exam might put more 
effort into their other courses, resulting in uncharacteristically low scores on the final ex- 
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am. Alternatively, students who had not performed as well on prior exams, but who still 
wanted to improve their course grade, could be expected to show relative improvement 
on the final exam. Fortunately, the data give some insights into this phenomenon. 

First consider the exam scores for the students who earned 100 percent on at least one of 
the first two exams. On average, these students scored 93.8 percent on the first two exams 
but only 81.5 percent on the final exam. Hence, students who had scored the highest on 
the earlier exams dropped 12.3 points on the final exam to just a few points above the 
class average. At the same time, of the students who earned a 100 percent on the final 
exam, none had earned a 100 on an earlier exam. In fact, these students had averaged on¬ 
ly a 78.8 percent on the first two exams, below the average. Both of these results support 
the proposition that external end-of-semester issues were at play. As an instructor, it is 
particularly satisfying to see students who had struggled on earlier material rise up to 
achieve on the final exam. 

Figure 1. Data for hypothesis HI: performance on exams will increase with the use 
of randomized homework assignments. 

Exam 1 (Forecasting) Exam 2 (SPC) Exam 3 (Inventory) 

pre-random post-random pre-random post-random pre-random post-random 


Mean 

77.47 


81.30 

76.39 


80.02 

76.10 


78.75 

Variance 

304.9 


255.0 

118.0 


208.5 

385.1 


223.7 

Obs. 

63 


94 

63 


92 

60 


90 

t-statistic 


- 1.39 



- 1.79 



- 0.89 


p-value 


0.083 



0.038 



0.188 



The second hypothesis, H2, measured student performance on the homework assignments 
where a performance decrease during the randomization phase is anticipated. In Figure 2, 
it is seen that homework scores decreased by 2.96 points (p-value = .053) on the first 
homework assignment, forecasting; was lower by 9.35 points (p-value = .000) on the 
second assignment, SPC; and yet increased by 2.18 points on the third assignment, inven¬ 
tory. The magnitude of the impact, in terms of Hake’s (1998) statistic, g, is moderate for 
the forecasting (g = -0.48) and inventory (g = 0.37) and is strong for SPC (g = -1.57). 

Figure 2. Data for hypothesis H2: performance on homework assignments will de¬ 
crease with the use of randomized homework assignments. 

HWl (Forecasting) HW2(SPC) HW 3 (Inventory) 

pre-random post-random pre-random post-random pre-random post-random 


Mean 

93.89 


90.93 

94.03 


84.68 

94.03 


96.21 

Variance 

89.1 


179.8 

77.0 


258.9 

143.4 


64.6 

Obs. 

63 


94 

63 


92 

60 


90 

t-statistic 


1.63 



4.65 



- 1.24 


p-value 


0.053 



0.000 



0.609 



Further review of the data indicates that much of this impact comes from a change in the 
rate of perfect homework scores after randomization. The rate of perfect scores decreased 


The Journal of Effective Teaching, Vol. 11, No. 2, 2011, 4-17 

© 

2011 All rights reserved 







Berardi 


14 


by 15.3 percent on the forecasting assignment, by 49.7 percent on SPC, while perfect 
scores increased by 12.2 percent on the inventory homework. 

To understand the implication of this, consider the impact randomized values had on the 
scores. For SPC in particular, it is apparent that many of the perfect scores on the home¬ 
work during the static data phase were the result of copying by students; simply giving 
each student unique values cut the rate of perfection in half. In the randomization phase, 
however, this means more students did not have a high score on the SPC homework, arti¬ 
ficially propping up their average. As a result, doing well on the final homework assign¬ 
ment, inventory, became more important and students responded. So not only is the ap¬ 
parent rate of cheating decreased, but motivation for students to perform on the next as¬ 
signment appears increased. While these results are encouraging for the class as a whole, 
viewing homework’s effect on exam performance from an individual student perspective 
is more refined and discussed next. 

The third hypothesis, H3, tested individual student performance gaps on homework com¬ 
pared to the related exam problems. Figure 3 shows a reduction in the gap between 
homework and exam performance on all three assignments. On the first exam, forecast¬ 
ing, the performance gap between the static data group and random data group is reduced 
by 6.8 points, from a gap of -16.4 to -9.6 (p-value = .006). On exam two, SPC, the differ¬ 
ence is reduced by 13.4 points, from -17.6 to -4.2 points (p-value = .000). The third ex¬ 
am, inventory, shows a very small performance gap of 0.5 points, from a gap of -17.9 to - 
17.4 points (/;-value = .438). Therefore, on the first two assignments, a significant statis¬ 
tical and practical reduction in individual perfonnance gaps on homework and exam 
scores is achieved. On the third exam, end-of-semester gaming, as discussed under HI, 
appears to play a role in the near zero change in performance gap. 

Figure 3. Data for hypothesis H3: individual performance gaps on exams and 
homework assignments will decrease with the use of randomized homework assign¬ 
ments. 

Forecasting (HW 1-Exam 1) SPC (HW 2- Exam 2) Inventory (HW 3-Exam 3) 

pre-random post-random pre-random post-random pre-random post-random 


Mean 

- 16.4 


- 9.6 

- 17.6 


- 4.2 

- 17.9 


- 17.4 

Variance 

236.8 


322.4 

127.9 


301.0 

405.8 


231.8 

Obs. 

63 


94 

63 


92 

60 


90 

t-statistic 


- 2.53 



- 5.84 



- 0.16 


p-value 


0.006 



0.000 



0.438 



The improvements related to the forecasting and SPC material are considerable and war¬ 
rant added discussion. Notice for the control group, the decline in exam performance on 
the forecasting and SPC material is 16.4 points and 17.6 points, respectively. This means 
that a student with static data who gets a homework score in the high 90s could struggle 
to achieve a low 80s score on the same material under exam conditions. While one could 
certainly expect to see some decrease given the control and pressure inherent in an exam 
situation, any instructor would be concerned by this magnitude of effect. 
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Once again, implementing randomized homework values had a significant practical im¬ 
pact in addition to a statistical one. On the forecasting material, the homework versus ex¬ 
am performance gap was reduced by more than 41 percent to a 9.6 point decline for the 
students receiving randomized homework values. For SPC, the perfonnance gap was re¬ 
duced by nearly 76 percent to a satisfyingly small 4.2 point difference. In other words, 
with randomized data the homework assignments have become better indicators of a stu¬ 
dent’s ability to perform on the exam. And since overall exam performance increased per 
HI, student learning, as measured by performance on the same material at a later point in 
time under controlled conditions, appears improved. 

Conclusion and Future Work 

While the design used in this study was necessarily tailored to the issues as they pertained 
to homework in an established operations management course, the randomized home¬ 
work approach and lessons learned should be applicable to other quantitative courses. In 
this study, exam scores increased when using randomized homework values compared to 
using static values, thereby supporting hypothesis HI. Student homework scores de¬ 
creased as postulated in H2, where much of this drop was due to a lower rate of perfect 
scores on homework after randomization. On one assignment, perfect scores dropped by 
nearly 50 percent, which indicates that in the static data phase many students were proba¬ 
bly just copying answers. Finally, with randomized values, homework scores became a 
more accurate reflection of student ability as the gap between homework and exam scores 
was reduced per hypothesis H3. 

Randomized values are not a panacea, though, as external factors affecting student per¬ 
formance can be significant. In particular, end-of-semester concerns by students who are 
trying to maximize their grades across several courses are not trivial. On the positive side, 
it was seen that the students who perform best on the final exam were those who strug¬ 
gled near the class average earlier in the semester. At the same time, high performing 
students may still suffer a drop-off as they focus their energies on other courses. Investi¬ 
gating the impact of semester timing on performance by randomizing topic sequencing 
would be a significant study to conduct in itself. To do so in a meaningful timeframe 
would require having several sections of a course offered simultaneously so assignment 
timing can be randomized, which is more than a small campus such as mine affords. 

The randomized capabilities developed for this study or those available from publisher 
offerings may have even greater value for use in fully online courses. In addition to 
homework, problems presented on exams can include randomized data where values, and 
indeed problem structure, can be varied for each student. As in this current investigation, 
there are implications for both improving student learning and reducing academic dis¬ 
honesty in online sections. This is especially important given that exams in many online 
courses are completed in an un-proctored environment similar to what is seen with 
homework from traditional courses. I have just begun teaching a fully online section of 
operations management and I am collecting data on the initial phase of an investigation. 
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